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Abstract 

Optimal sensor scheduling with applications to networked estimation and control systems 
is considered. We model sensor measurement and transmission instances using jumps between 
states of a continuous-time Markov chain. We introduce a cost function for this Markov chain as 
the summation of terms depending on the average sampling frequencies of the subsystems and 
the effort needed for changing the parameters of the underlying Markov chain. By minimizing 
this cost function through extending Brockett's recent approach to optimal control of Markov 
chains, we extract an optimal scheduling policy to fairly allocate the network resources among 
the control loops. We study the statistical properties of this scheduling policy in order to 
compute upper bounds for the closed-loop performance of the networked system, where several 
decoupled scalar subsystems are connected to their corresponding estimator or controller through 
a shared communication medium. We generalize the estimation results to observable subsystems 
of arbitrary order. Finally, we illustrate the developed results numerically on a networked system 
composed of several decoupled water tanks. 

1 Introduction 

1.1 Motivation 

Emerging large-scale control applications in power grids [I], smart infrastructures |2j, intelligent 
transportation systems [3] , and aerospace systems [4] , are typically implemented over a shared com- 
munication medium. Figure [T] illustrates an example of such a networked system, where L decoupled 
subsystems are connected to their subcontrollers over a wireless communication network. A set of 
sensors in each subsystem sample its state and transmit the measurements over the wireless network 
to the corresponding subcontroller. Then, the subcontroller calculates an actuation signal (based 
on the transmitted observation history) and directly applies it to the subsystem. Unfortunately, 
traditional digital control theory mostly results in conservative networked controllers because the 
available methods often assume that the sampling is done periodically with a fixed rate [5][6] . When 
utilizing these periodic sampling methods, the network manager should allocate communication 
instances (according to the fixed sampling rates) to each control loop considering the worst-case 
possible scenario, that is, the maximum number of active control loops. In a large control system 
with thousands of control loops, fixed scheduling of communication instances imposes major con- 
straints because network resources are allocated even if a particular control loop is not active at the 
moment. This restriction is more evident in ad-hoc networked control systems where many control 
loops may join or leave the network or switch between active and inactive states. Therefore, we need 
a scheduling method to set the sampling rates of the individual control loops adaptively according 
to their requirements and the overall network resources. We address this problem in this paper by 
introducing an optimal stochastic sensor scheduling scheme. 

1.2 Related Studies 

In networked control systems, communication resources need to be efficiently shared between multi- 
ple control loops in order to guarantee a good closed-loop performance. Despite that communication 
resources in large networks almost always are varying over time due to the need from the indi- 
vidual users and physical communication constraints, the early networked control system literature 
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Figure 1: An example of a networked control system (left) together with flow diagram of the 
continuous-time Markov chain used for modeling the proposed stochastic scheduling policy (right). 



focused on situations with fixed communication constraints; e.g., bit-rate constraints [7-10 and 
packet loss 11-14 . Only recently, some studies have targeted the problem of integrated resource 
allocation and feedback control; e.g. 
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The problem of sharing a common communication medium or processing unit between several 
users is a well-known problem in computer science, wireless communication, and networked con- 
trol 
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For instance, the authors in |21 proposed a scheduler to allocate time slots between 
several users over a long horizon. In that scheduler, the designer must first manually assign shares (of 
a communication medium or processing unit) that an individual user should receive. Then, each user 

The 
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achieves its pre-assigned share by means of probabilistic or deterministic algorithms 
authors in 23f24 proved that implementing the task with the earliest deadline achieves the optimum 
latency in case of both synchronous and asynchronous job arrivals. In 25 , a scheduling policy based 
on static priority assignment to the tasks was introduced. Many studies in communication literature 
have also considered the problem of developing protocols in order to avoid the interference between 
several information sources when using a common communication medium. Examples of such pro- 
tocols are both time-division and frequency-division multiple access 26 27 . Contrary to all these 



studies, in this paper, we automatically determine the communication instances (and, equivalcntly, 
the sampling rates) of the subsystems in a networked system based on the number of active control 
loops at any given moment. We use a continuous-time Markov chain to model the optimal scheduling 
policy. 

Markov chains are very convenient tools in control and communication 



28 29 . Markov jump 



linear systems with underlying parameters switching according to a given Markov chain has been 
studied in the control literature 30 -33 . The problem of controlled Markov chains has always been 

an explicit solution to the problem of 



actively pursued 34-37 



In a recent study by Brockett 
optimal control of observable continuous-time Markov chains for a class of quadratic cost functions 
was presented. In that paper, the underlying continuous-time Markov chain was described using the 
so-called unit vector representation |38[|39| . Then, the finite horizon problem and its generalization 
to infinite horizon cost functions were considered. We extend that result to derive the optimal 
scheduling policy in this paper. 

the authors developed a stochastic sensor scheduling policy using Markov chains. 



In the study 40 



Contrary to this paper, they considered a discrete-time Markov chain to get a numerically tractable 
algorithm for optimal sensor scheduling. The algorithm in [40] uses one of the sensors at each time 
step while here, the continuous-time Markov chain can rest in one of its states to avoid sampling 
any of the sensors. Furthermore, the cost function in 40 was not written explicitly in terms of 
the Markov chain parameters, but instead it was based on the networked system performance when 
using a Markov chain for sampling the sensors. However, our proposed scheduling policy results in a 
separation between designing the Markov chain parameters and networked system, which enables us 
to describe the cost function needed for deriving our optimal sensor scheduling policy only in terms 
of the Markov chain parameters. 
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1.3 Main Contributions 



The objective of the paper is to find a dynamic scheduling policy to fairly allocate the network re- 
sources between the subsystems in a networked system such as the one in Figure [l](left). Specifically, 
we employ a continuous-time Markov chain for scheduling the sensor measurement and transmission 
instances. We use time instances of the jumps between states of this continuous-time Markov chain 
to model the sampling instances; i.e., whenever there is a jump from an idle state in the Markov chain 
to a state that represent a subsystem in the networked system, we sample that particular subsystem 
and transmit its state measurement across the shared communication network to the corresponding 
subcontroller. Figure [T] (right) illustrates the flow diagram of the proposed Markov chain. Every time 
that a jump from the idle node / to node St, 1 < I < L, occurs in this continuous-time Markov chain, 
we sample subsystem £ and send its state measurement to subcontroller £. The idle state / helps 
to tune the sampling rates of the subsystems independently. As an approximation of the wireless 
communication network, we assume that the sampling and communication are instantaneous; i.e., 
the sampling and transmission delays are negligible in comparison to the subsystems response time. 
We still want to limit the amount of communication per time unit to reduce the energy consumption 
and network resources. 

We mathematically model the described continuous-time Markov chain using unit vector rep- 
resentation 38 39 . We introduce a cost function that is a combination of the average sampling 
frequencies of the subsystems (i.e., the average frequency of the jumps between the idle state and 
the rest of the states in the Markov chain) and the effort needed for changing the scheduling policy 



(i.e., changing the underlying Markov chain parameters). We expand the results presented in 38 
to minimize the cost function over both finite and infinite horizons. Doing so, we find an explicit 
minimizcr of the cost function and develop the optimal scheduling policy accordingly. This pol- 
icy fairly allocates sampling instances among the sensors in the networked system. The proposed 
optimal scheduling policy works particularly well for ad-hoc sensor networks since we can easily 
accommodate for the changes in the network configuration by adding an extra state to the Markov 
chain (and, in turn, by adding an extra term to the cost function) whenever a new sensor becomes 
active and by removing a state from the Markov chain (and, in turn, by removing the corresponding 
term from the cost function) whenever a sensor becomes inactive. The idea of dynamic peer par- 
ticipation (or churn) in peer-to-peer networks have been extensively studied in the communication 
literature 41 42). However, not much attention has been paid to this problem for networked control 
and estimation. 

Later, we focus on networked estimation as an application of the proposed stochastic sensor 
scheduling policy. We start by studying a networked system composed of several scalar subsystems 
and calculate an explicit upper bound for the estimation error variance as a function of the statistics 
of the measurement noise and the scheduling policy. The statistics of the scheduling policy are 
implicitly dependent on the cost function. Hence, we can achieve the required level of performance 
by finely tuning the cost function design parameters. We generalize these estimation results to 
higher-order subsystems when noisy state measurements of the subsystems are available. In the case 
where noisy output measurements of the subsystems are available, we derive an estimator based on 
the discrete-time Kalman filter and calculate an upper bound for the variance of its error given a 
specific sequence of sampling instances. Lastly, we consider networked control as an application of 
the proposed sensor scheduling policy. We assume that the networked control system is composed of 
scalar subsystems that are in feedback interconnection with impulsive controllers (i.e., controllers that 
ideally reset the state of the system whenever a new measurement arrives) . We find an upper bound 
for the closed-loop performance of the subsystems as a function of the statistics of the measurement 
noise and the scheduling policy. We generalize this result to pulse and exponential controllers. 



1.4 Paper Outline 

The rest of the paper is organized as follows. In Section [2j we introduce the optimal stochastic 
scheduling policy and calculate its statistics. We apply the proposed stochastic scheduling policy 
to networked estimation and control systems in Sections [3] and [4j respectively. In Section [5j we 
illustrate the developed results numerically on a networked system composed of several decoupled 
water tanks. Finally, we present the conclusions and directions for future research in Section [6j 
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1.5 Notation 



The sets of integer and real numbers are denoted by Z and R, respectively. We use O and E to denote 
the sets of odd and even numbers. For any n€Z and x € K, we define Z > (>)„ = {m ^7L\m> (>)n} 
and M > (>) ;r = {y 6 K | y > (>)x}, respectively. We use calligraphic letters, such as .4 and A", to 
denote any other set. 

We use capital roman letters, such as A and C , to denote matrices. For any matrix A, Ai denotes 
its i-th row and denotes its entry in the i-th row and the j-th column. 

Vector &i denotes a column vector (where its size will be defined in the text) with all entries equal 
zero except its i-th entry which is equal to one. For any vector x € R™, we define the entry- wise 
operator x 2 = {x\ ■ ■ ■ x^] 7 . 



2 Stochastic Sensor Scheduling 

In this section, we develop an optimal stochastic scheduling policy for networked systems, where 
several sensors are connected to the corresponding controllers or estimators over a shared commu- 
nication medium. Let us start by modeling the stochastic scheduling policy using continuous-time 
Markov chains. 



2.1 Sensor Scheduling Using Continuous-Time Markov Chains 

We employ continuous-time Markov chains to model the sampling instances of the subsystems. 
To be specific, every time that a jump from the idle node I to node St, 1 < £ < L, occurs in 
the continuous-time Markov chain described by the schematic flow diagram in Figure [I] (right), we 
sample subsystem £ . We use unit vector representation to mathematically model this continuous-time 



Markov chain 38 39 



We define the set X = {e\, e2, . . . , e„} C I" where n = L + 1. The Markov chain state x(t) € 
K™ takes value from X, which is the reason behind naming this representation as the unit vector 
representation. We associate nodes Si, S2, • ■ • , Sx , and I in the Markov chain flow diagram with 
unit vectors ex, €2, . . ■ , e^, and e n , respectively. Following the same approach as in 39 , we can 
model the Markov chain in Figure [I] (right) by the Ito differential equation 

dx(t) = (G' in x(t) dN' n (t) + G' ne x(t) dN^(t)) , (1) 

where {N l ne (t)} teR>g and {N' (n (t)}teR >0) 1 < £ < L, are Poisson counter processes^ with rates \ n e(t) 
and Xi n (t), respectively. These Poisson counters determine the rates of jump from Si to /, and vice 
versa. In addition, we have G' ln = (eg — e„)ej and G' nl = (e„ — ei)ej , 1 < £ < L. Let us define 
m = 2L. Now, we can rearrange the Ito differential equation in ([!]) as 

m 

dx(t) = Y,G l x{t)dN t {t), (2) 

i=l 

where {Ni(t)} t ^m >0 , 1 < i < m, is a Poisson counter process with rate denoted as 

/*(*) = ! ^L«-i)/2J+i(*)i *eO, ( } 

P V ' \ A L(i _ 1)/a j + i,„(t), ieE, y ' 

and 

r< - J G n,L( 4 -i)/2j+i' 1 e °' u\ 

\ U L(«-1)/2J+I,n' fc 

The Poisson counters {^Vi(^)}tgR >0 , 1 < i < m, determine the rates of jump between the states 
of the Markov chain in ([2]). Now, noting that this Markov chain models the sampling instances 

1 Recall that a Poisson counter N(t) is a stochastic process with independent and stationary increments that 
starts from zero N(Q) = 0. Additionally, ¥{N(t + Ai) - N(t) = k} = (// +A ' X(t)dt) k exp(/ t t+At A(t)dt)/fc! for any 
t,At G M>0 and k £ Z>q. In the limit, when replacing At with At and AiV(t) = N(t + At) — N(t) with dN(t), we get 
P{diV(t) = 0} = 1 - \(t)dt, ¥{dN(t) = 1} = \(t)dt, and ¥{dN(t) = k} = for k e Z> 2 . For a detailed discussion on 
Poisson counters see [39||43] . 
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{T/}°^q, 1 < I < L, using the jumps that occur in its state x(t), we can control the average sampling 



frequencies of the sensors through the rates /ij(t), 1 < i < m. Similar to 38 , we assume that we can 
control the rates as 

m 

= Mi,o + (5) 

3=1 

and thereby control the average sampling frequencies. Control signals Uj(t), 1 < j < m, are chosen 
in order to minimize the cost function 

J = r lim o E|i^ ^^e^(i)d7V 2£ (t)+w(<) T w(t)d<|, (6) 

where & € R>o, 1 < I < L, are design parameters. Note that the cost function consists of two 
types of terms: y eJ l x(t)&N2t{t) denotes the average frequency of the jumps from / to Si in the 

Markov chain (i.e., the average sampling frequency of sensor t); and ^ f? u(t) T u(t)dt penalizes the 
control effort in regulating this frequency If the latter term is removed, the problem would become 
ill-posed as the optimal rates Hi(t) then is zero and M{dNi(t)} = 0. Consequently, the average 
sampling frequencies of the sensors vanish. 

Considering the identity EjdiV^i)} = (^21,0 +2j=i a 2£,jUj(t))dt, we can rewrite the cost func- 
tion in Q as 

J= lim e(- / c T x{t)+u(t) T Sx{t) + u(t) T u(t)dt\, (7) 



T^oo [T J Q 

where c = e„ J2e=i Ce^2t.,o an d S £ ]R mXTl is a matrix whose entries are defined as Sji = J2e=i ii a 2i,j 
if i = n and sji — otherwise. In the next subsection, we find a policy that minimizes ^ with 
respect to the rate control law ^ and subject to the Markov chain dynamics pi). Doing so, we 
develop an optimal scheduling policy which fairly allocates the network resources (i.e., the sampling 
instances) between the devices in a sensor network. 

2.2 Optimal Sensor Scheduling 

We start by minimizing the finite horizon version of the cost function in ([6|. The proof of the 



following theorem is a slight generalization of Brockett's result in 38 but follows the same line of 
rin£j 



reasoning 



Theorem 2.1 Consider a continuous-time Markov chain evolving on X = {e\, . . . ,e n } C M™, 
generated by M). Let us define matrices A = YllLi Mi.Q^i an d B% — Sj=i 'j > where for all 
1 < j < nn, Gi and ctij are introduced in and respectively. Assume that, for given T € R>o 
and c : [0,T] — ¥ M. n , the differential equation 

rn 

k(t) = -c(t) - A T k(t) + - J2(Sj + Bjk(t))- 2 ; k(T) = k f , (8) 

i=i 

has a solution on [0,T] such that, for each (t,x) S [0,T] x X , the operator A — Y^lLi + 
SijxBi is an infinitesimal generator. Then, the control law 

u t (t,x) = -l ) (k(t) T B l + S l )x(t), l<i<m, (9) 



minimizes 



J = E I - y c(t) 1 + u(i) 1 Sx(t) + u(t) 1 w(t)d< + -k) x{T) 
Furthermore, J = ^fc(0) T E {x(0)}. 



2 The statement makes use of the concept of infinitesimal generators. See [44 pp. 124] for definition and discussion. 
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Proof: We follow a similar reasoning as in 38! to calculate the optimal Poisson rates. By adding 
and subtracting the term fc(£) T E{a;(i)}| from the right hand-side of the scaled cost function TJ — 
k]E{x{T)}, we get 



TJ — kjE {x(T)} = E |^ 



c{t) T x(t) + u(t) T Sx(t) + u(t) T u(t)dt 



k(t) T E{x(t)}\ + k(t) T E{x(t)}\ 



(10) 



■ E ■ 



c(t) T x(t) + u(t) T Sx(t) + u(t) T u(t)dt 



Using the identity k(t) T E{x(t)}\^ = E | d(k(t), x(t)) X inside (10), we get 



TJ - kjE {x(T)} = - k(t) T E {x(t)}\ 



E 



d(k(t),x(t)) 



(11) 



E ■ 



c(t) T x(t) + u(t) T Sx(t) + u(t) T u(t)dt 



Using Ito's Lemma 44 p. 49], we know that 

m 

d(k(t),x(t)) = (k(t),x(t))dt + Y / (Kt),G l x(t) dNi(t)), 



which transforms (11) into 



TJ - kjE {x(T)} = - k{t) T E {x(t)}\l + E < / (k(t),x(t))dt + ^{k(t), Gix(t) dN^t)) 



i=i 



E • 



c(t) T x(t) + u(t) T Sx{t) + u{t) T u(t)dt 



Taking expectation over x(t) and the Poisson processes {Ni(t)} te R >0 , 1 < i < m, we get 



TJ -k}E{x(T)} = - k(t)*E{x(t)}\ + / (k(t) +c(t) +A 1 k{t),p(t))dt 

Jo 

+ EL(t) T u(t)+jTu l (t){S l x(t) + (k(t),B l x(t)))dt\, 



(12) 



where, for 1 < i < m, Si is i-th row of matrix S and p(t) = E{x(t)}. We can rewrite (12) as 
TJ - k]E{x(T)} = / (k(t) + c(t) + A T k(t) - - Y^{Sj + Bjk{t))- 2 ,p{t))dt 

Jq i=l 
_ ( pT m 

- k(t) T E{x(t)}\ Q +E / Yl 



Ui {t) + -(k(t) 1 Bi + Si)x(t) 



(13) 



using completion of squares. As there exists a well-defined solution to the differential equation ([8]), 
the first integral in ( 13 ) vanishes. Hence, the optimal control law is given by ^ since this control 
law minimizes the last term of (13). Consequently, equation (13) gives 

TJ = kjE {x(T)} ~ k(t) T E {x(t)}\l = fc(0) T E{a;(0)} . 
This completes the proof ■ 



Based on Theorem 2.1, we are able to solve the following infinite- horizon version of the optimal 
scheduling policy. 



G 



Corollary 2.2 Consider a continuous-time Markov chain evolving on X = {e±, . . . , e n } C K n , 
generated by Q). Let us define matrices A = Mi O^j an d — Y^j=\ a ijGj> where for all 

1 < hJ ' < Tij G.j and Qijj are introduced in ^ and |5p, respectively. Assume that, for a given 
c S K n , £/ie nonlinear equation 

o 



' A T 


1 ' 




fc 


1 


_ 1 T 







8 


~ 4 



an infinitesimal generator. Then, the control law 





— c 







' A - 


" Z^z=l 







minimizes 



J = lim E 

T^co 



T 



c T x(i) + u(t) T Sx(t) + u(t) T u(t)dt 



Furthermore, we have J = g. 

Proof: Since x(t) € A" is bounded (because ||x(£)||2 = 1 for i G K>o), we get the identity 



lim E I - , 



c T a;(t) + u(t) T Sx(t) + u(t) T u(t)dt 



= lim E 

T-s-oo 



T 



c T x(t) + u(t) T Sx{t) + u{t) T u{t)dt + -kJx(T) 



According to Theorem 2.1 in order to minimize (15 1 for any fixed T S K>o, we have 



-. m 

k(t) = -c(t) A T k(t) + ]T(S S T + Bjk(t)\\ 



i=l 



(14) 



(15) 



(16) 



with the final condition k(T) — k . Defining q(t) — k(T — t) — k — git, we get 
q(t) = - fc(T-i) - gl 

1 m 

= A T k(T - t ) + c-~J2(Sj + Bjk(T t))- 2 gl 

^ m 

= A T (q(t) + k + git) + c- gl - - J2( S ? + + + ^)\ 2 ■ 

i=l 

Note that A T 1 = and Bjl = 0, 1 < i < m, as A and are infinitesimal generators. Hence, 

^ m 

q(t) = A T (q(t) +k )+C-Ql-- ^(Sj + Bj(q(t) + k Q )\ 2 
1 m 1 m 



(17) 



Notice that q* = is an equilibrium of ([17]), so g(£) = for all t £ [0, T] since g(0) = k(T) - k = 0. 
Therefore, we get fc(i) = fco + £>1(T — t), which results in ^(k(t) T B t + Si) = \{kj) Bi + Si), since 
1 T Bi = 0, 1 < i < m. As a result, when T goes to infinity, the controller which minimizes (15) is 
given by (14). Furthermore, we have 

J= lim -k{0) T E{x(0)} = lim - (fc + gl(T - 0)) T E{x(0)} = gl T E{x(0)} = g. 



This concludes the proof. 
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Corollary |2.2| introduces an optimal scheduling policy to fairly allocate measurement transmissions 
among sensors according to the cost function in ([6]). By changing the design parameters £f , 1 < I < L, 
we can tune the average sampling frequencies of the subsystems according to their performance 
requirements. In addition, by adding an extra term to the cost function whenever a new subsystem in 
introduced or by removing a term whenever a subsystem is detached, we can easily accommodate for 
dynamic changes in an ad-hoc network. In the remainder of this section, we analyze the asymptotic 
properties of the optimal scheduling policy in Corollary |2.2| 



2.3 Average Sampling Frequencies 

In this subsection, we study the relationship between the Markov chain parameters and the effective 
sampling frequencies of the subsystems. Recalling from the problem formulation, {Tf}°l Q denotes 
the sequence of time instances that the state of the Markov chain in ([I]) jumps from the idle node I 
to Si and hence, subsystem I is sampled. Mathematically, we define these time instances as 

T £ = inf{t > | 3 e > : x(t - e) = e n A x(t) = ej, 

and 

T/ +1 = ia£{t > T/ I 3 e > : x(t - e) = e n A x(t) = e e }, i e Z> . 

Furthermore, we define the sequence of random variables {Af}°^ such that Af = T/ +1 — Tf for 
all i £ Z>o- These random variables denote the time interval between any two successive sampling 
instances of sensor £. We make the assumption that the first and second samples happen within 
finite time almost surely: 

Assumption 2.3 P{T^ < oo} = 1 and P{Tf < oo} = 1. 

This assumption is not restrictive. Note that it is trivially satisfied if the number of subsystems 
is finite, the Markov chain is irreducible, and the rates of Poisson processes are finite and uniformly 
bounded away from zero. 

Lemma 2.4 {A^}°^ are identically and independently distributed random variables. 

Proof: According to the Markov property [44j p. 117], we know that, for a given x(Tf), the 
trajectory {x(t) \ t > Tf} is independent of the history {x(t) 1 1 < Tf}. Noting that x{T-) — e^ for all 
i > 1, gives that {Af}°l are independent random variables. In addition, the Markov chain in ^ 
and the control law in (14) are time invariant. Therefore, the closed-loop Markov chain is also time 
invariant, and as a result, {Af}°SL have equal probability distributions. ■ 

Now, we are ready to prove that the average sampling frequency of subsystems £ is actually equal 
to lim<r_ i . 00 ^ Jq eJ l x{t)dN2i{t) . However, first, we prove the following useful lemma. 



Lemma 2.5 Let the sequence of sampling instances {T{}°^ satisfy Assumption 2.3 Then, 

hm — = — ——. (18) 
t^oo t E{Af} v ' 

where Ml = max {i > 1 | T[ < counts the number of jumps prior to any given time t g M>o and 
x = y means that F{x = y} = 1. 



Proof: This proof follows a similar reasoning as in the proof of Theorem 14 in 45 . For any given 
€E Z >0 , we have 

i=l i=l 

Note that since P{Tq < oo} — 1 according to Assumption |2.3[ we get lim^oo Tq/i3 = 0. Therefore, 
we have 

lim tf a d hm TAt,. 

i=l 
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Notice that P{Aq = T-f — Tjf < 00} = 1 according to Assumption 2.3 Therefore, using Lemma 2.4 
we get P{Af < 00} = 1 for all i € Z> . Consequently, E{|Af|} = E{Af} < 00. Now, using the 
strong law of large numbers |46| , we get 



lim ^ = E{Aj}. 



(19) 



For any t G M> , we have T^ t < t < T l M , +v Therefore, we get T^/M| < t/Mf < T^ e+1 /Mf. 

Notice that limj-^oo M e { = 00 since, as proved earlier, P{Af < 00} = 1 for all i € Z>o- Using (19), 
we get 



T 



lim 

t-s-oo Mi 



rpt 

M t ' ,. Mf + 1 



E{A*} 



lim 



- E{Af}, 



which results in (18 1. 



We now state our main result concerning the average sampling frequency of the sensors denoted 



by 



ft = lim E • 

T-s-oo 



elx{t) dN 2e (t) \, 1 < £ < L. 



Theorem 2.6 Let the sequence of sampling instances {Tf }*jZ satisfy Assumption 2.3 Then, the 
average sampling frequency of sensor £ is equal to 



E{Aj} 

where p(t) = E{x(t)} can be computed by 



1 m \ 
i=i J 



p{t) =U-~f^ BiAikjB, + sA p(t), p(0) = E {x(0)} , 



(20) 



with notation A(kJ Bi + Si) = diag((fcg + Si)ei, . . . , (kj Bi + Si)e n ). 

Proof: The first part of the proof is a direct consequence of Lemma 
Jg T eJ l x{t)AN2i{t) . For the second part, we can compute p(t) using 



2.5 



and the fact that Mk 



p(t) = Ap(t) + E i J2 u ^ x(t))B lX (t) \ , p(0) = E {x(0)} 



(21) 



. i=l 



Substituting (14) inside (21), we get 



1 f m 

>(t) = Ap(t) - -E <^ ^(fc^S* + Si)x{t)B iX {t) 
= Ap(t) - ^E <| j[> T A + 5<) 



" !Cl(t) " 












*n(*) 




. ar n (*) 



4p(t) - -E <( 



x n(t) Y!j = l( k Q B i + S^CjX^t) 



Note that x^(<) E?=i (kg Bj + Sj)ejXj (t) = (kj Bi + Si)e(X^(t) for 1 < £ < n, since x(t) G A" is a unit 



1 )^2_ 

vector in E™. Therefore, we get p0|. Now, assuming that p(t) converges exponentially to a nonzero 
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steady-state value as time goes to infinity, we can expand the expression for the average sampling 
frequencies of the sensors as 

= T i ^o E l h j G ™ X ^ (^ 2e >° a 2tA k B i + Sj)^)] dt 

1 f T ( 1 m \ 

= T lin ^ ji e^p(t) I M2£,o - 2 2J a ^,ji k o B j + S j) e nj dt 

1 m 1 

3=1 / 

where the third equality follows again from the fact that x(t) £ X is a unit vector. 



Theorem |2.6| allows us to calculate the average sampling frequencies of the subsystems. We use 
these average sampling frequencies to bound the closed-loop performance of the networked system 
when the proposed optimal scheduling policy is implemented. 



3 Applications to Networked Estimation 

In this section, we study networked estimation based on the proposed stochastic scheduling policy. 
Let us start by presenting the system model and the estimator. As a starting point, we introduce a 



networked system that is composed of scalar decoupled subsystems. In Subsections 3__3 and |3.4[ we 
generalize some of the results to decoupled higher-order subsystems. 



3.1 System Model and Estimator 

Consider the networked system illustrated in Figure [T] where subsystem I, 1 < t < L, is a scalar 
stochastic system described by 



dz e {t) = -7<2/(t) dt + a e dw e (t); z e (0) = 0, (23) 

with given model parameters ^e.,cri £ M>o- Note that all subsystems are stable. The stochastic 
processes {wg(t)} t t£m >Q , 1 < I < L, are statistically independent Wiener processes with zero mean. 
Estimator i receives state measurements {yf }°^ at time instances {Tf }^ > sucn that 

y\ = ztpf) + ni; Vz £ Z> , (24) 

where {nf}fl denotes measurement noise sequence, which is composed of independently and iden- 
tically distributed Gaussian random variables with zero mean and specified standard deviation rjg. 
Let each subsystem adopt a simple estimator of the form 

±z e (t) = -im(t); k{T t i )=vl (25) 

for t £ [Tl,T- +1 ). We define the estimation error eg(t) = zg(t) — zg(t). Estimator I only has 
access to the state measurements of subsystem I at specific time instances {T/}^ but is supposed 
to reconstruct the signal at any time t £ M>o- Notice that this estimator is not optimal. In 
Subsection |3.4[ we will consider estimators based on Kalman filtering instead. 



3.2 Performance Analysis: Scalar Subsystems 

In this subsection, we present an upper bound for the performance of the introduced networked 
estimator. First, we prove the following simple lemma. 
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Lemma 3.1 Let the function g : R>o — > K be defined as g(t) = c\e~ 2lt + |^(1 — e~ 27 *) with given 
scalars ci,c 2 £ K and 7 £ R>o such that 2^c\ < c 2 . Then, 

(a) g is a non- decreasing function on its domain; 

(b) g is a concave function on its domain. 

Proof: For part (a), note that if 2^c\ < C2, the function g{t) is continuously differentiable and 
dg(t)/dt = —(27c! — c 2 )e~ 27 ' > for all t £ R>o- Hence, g(t) is a non-decreasing function on its 
domain (since it is continuous). On the other hand, for part (b), note that if 2jci < C2, the function 
g(t) is double continuously differentiable and d 2 g(t)/dt 2 = 27(2701 — C2)e~ 27 * < for all t £ K>o- 
Therefore, g(t) is a concave function on its domain. ■ 

The following theorem presents upper bounds for the estimation error variance for the cases where 
the measurement noise is small or large, respectively. 



Theorem 3.2 Assume that subsystem I, 1 < I < L, is described by (23) and let the sequence of 
sampling instances {T[}^Zq satisfy Assumption 2.3 Then, if wt < \J\j (2^fi)ai, the estimation error 
variance is bounded by 



E{ej(t)} < nje- 2 ^/^ + ^- fl - e^^) , 

27<> V / 

otherwise, ifrje > s /l/{2^ t )at, 



E{e 2 (i)} < 4 



2-fi 



1 - e 



-2-ye/h 



(26) 



(27) 



Proof: Using ito's Lemma 44 p. 49], for all t £ [T/,T/ +1 ), we get 
de € (t)= ' ' ] 



— z e (t) - j e e e (t) - jiZt(t)J dt + a e dwi(t) = -^ie t {t)dt + a e dw e (t), 

with the initial condition e^(T/) ~ —nf- First, let us consider the case where r\i < \J \j (2^1)0 1. 
Again, using Ito's Lemma, we get 



d(e?(*)) - {-2-yt ej(t) + o 2 )dt + 2e e {t)a e dwt(t), 



and as a result 



dt 



E{e 2 (t)|Af} = -27«E{e 2 (t)|Af} + a 



1 ' 



where E{e 2 (T/) | Af} = n 2 . Hence, for all t £ [Tf,Tf +1 ), we have 

E{ef(t)|Af} = r ? ?e- 2 ^ + ^-(l-e- 2 ^) 



Now, using Lemma [3. 1| (a), it is easy to see that 



E{e 2 (t) I Af} < + & (l _ e ~ 27 ^ 



Note that 



E{e 2 (t)}=E{E{e 2 (i)|Af}}<E In 2 



2n 



1-e 



(28) 



By using Lemma 3.1 (b) along with Jensen's Inequality |44j p. 320], we can transform (28) into (26). 
For the case where rji > ■Jl/(2'yi)<Ji, we can similarly derive the upper bound 



E{ e 2 (t)|Af}<, ? 2 + |^(l- e - 27 ^ 



which results in (27), again using Jensen's Inequality. 



Note that the upper bound (26) is tighter than (27) when the equality 77^ = ^l/{2 r yi)u£ holds. 
In the next two subsections, we generalize these results to higher-order subsystems. 
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3.3 Performance Analysis: Higher-Order Subsystems with Noisy State 
Measurement 

Let us assume that subsystem £, 1 < £ < L, is described by 

dzp(t) = Apzp(t)dt + H e dw e (t); z e {0) = 0, (29) 

where zg (t) £ R d * is its state with dp £ Z>i and Ap is its model matrix satisfying X(Ap + Aj) < 0. 
In addition, {wp(t)}t^R >0 > 1 < £ < L, is a, tuple of statistically independent Wiener processes with 
zero mean. Estimator I receives noisy state-measurements {yf}^ a * tmie instances {T/}°^ , such 
that 

yi = z t (T?) + 4; VzeZ> , (30) 

where {nf}°^ denotes the measurement noise and is composed of independently and identically 
distributed Gaussian random variables with E{nf } = and E{nfnf T } = Rp. We define the estimation 
error as ep(t) — zp(t) — zp(t), where for all t £ [T/,T/ +1 ), the state estimate zp(t) is derived by 

lfc(t) = AM*); zi(if) = yf, 

The next theorem presents an upper bound for the variance of this estimation error. For scalar 



subsystems, the introduced upper bound in (31) is equivalent to the upper bound in (27) 



Theorem 3.3 Assume that subsystem £, 1 < £ < L, is described by (29) and let the sequence of 



sampling instances {T[}°°^ satisfy Assumption 2.3 Then, the estimation error variance is bounded 
by 

E{|M;)I| 2 } < traced) + .^fff, (l - . (31) 

Proof: Using Ito's Lemma, for all t £ [T/,T/ +1 ), we get 

d||e £ (i)|| 2 = e e (t) T {A e + Aj)ep(t)dt + trace(iJ T H)dt + ep(t) T Hdw e (t) + dwp(t) T H T e t (t), 
and as a result 

^E{|Mt)|| 2 1 A^} = trace(if T if) + E{e e (t) T (A e + Aj)e t (t) | Af} 
< trace(ff T ff) + \{A e + A])E{\\e e (t)\\ 2 \ Af}, 

with the initial condition E{||e£(T/)|| 2 } = trace(i^). Now, using the Comparison Lemma 47 p. 102], 
we get 

E{|Mi)|| 2 | A|} < tracc^)^^ + (l - , 



for t £ [T/,T l+1 ). Using Lemma 3.1 and Jensen's Inequality, we get (31) 



It is possible to refine the upper bound (31) for the case where tr ace( _Ri ) < l/(2\X(Ap + Aj)\) 



trace(ff H), following a similar argument as in the proof of Theorem 



3.2 



3.4 Performance Analysis: Higher-Order Subsystems with Noisy Output 
Measurement 

In this subsection, we assume that estimator £, 1 < £ < L, receives noisy output measurements {yf }°^ n 
at time instances {T/}°^ , such that 

y i i =C e z i (T l i ) + ni; Vz £ Z> , (32) 

where Cp £ MP eXde (for a given output vector dimension pp £ Z>i such that pp < dp) and the mea- 
surement noise {n\ }°^ is a sequence of independently and identically distributed Gaussian random 
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variables with E{rif} = and E{nfnf T } = Rg. For any sequence of sampling instances {Tf}°l , we 



can discretize the stochastic continuous-time system in ( 29 1 as 

z e [i + 1] = P^j^M + Gf 
where zi[i] = z(Tf), Fg[i] — e A ^ Ti + 1 ~ Ti \ and the sequence {G^[i]}^ is chosen such that 

G < [i]G < [i] T = / e Ar HH T e A r dr, Vt € Z> . 

Jo 

In addition, {w£[i]}^ ^ s a sequence of independently and identically distributed Gaussian random 
variables with zero mean and unity variance. It is evident that yi[i] = CiZ^[i]+nj. We run a discrete- 
time Kalman filter over these output measurements to calculate the state estimates with 
error covariance matrix Pt[i] — E{(zi[i] — zg[i])(z(i[i] — zi[i]) T }. For inter-sample times t 6 [Tf, Tf +1 ), 
we use a simple prediction filter 

^(t) = A e z e (t); z e (Tf) = i e [{\. (33) 

Let us define the estimation error as e^(t) = zi(t) — Zi(t). The next theorem present an upper bound 
for the estimation error variance. 



Theorem 3.4 Assume that subsystem I, 1 < I < L, is described by (29). Then, the estimator given 
by \3£fy is an optimal mean square error estimator and for any fixed sequence of sampling instances 
{Tf}°^ , the estimation error is upper-bounded by 

n\W{t)\\ 2 1 Ttr ~ Tf} < traced) + ^f^i f 1 " ^ +A '^~^) ■ (34) 
Proof: First, note that for t € [Tf,Tf +1 ), the estimator 

±z t {t) = Atzt{t); z t (?i) = {z e (Tf) \y{,..., yj}, 

is an optimal mean square error estimator. This is in fact true since the estimator £ has not re- 
ceived any new information over [Tf,t] and it should simply predict the state using the best avail- 



able estimation {z e (Tf) \ y\, . . , , yf}. Now, recalling from 48 , we know that {Z(,{Tf) \ y\ , , . . , yf} = 
{zi[i] | yf, . . . , yf} = zt[i]. This completes the fi rst p art of the proof. For the rest, note that following 
a similar reasoning as in the proof of Theorem 3.3 for all t £ [Tf, Tf +1 ), we get 



^E{|Mi)|| 2 | A^} < trace(ff T fl) + \{A t + Aj)E{\\e t (t)\\ 2 | Af}, 
with the initial condition E{||e£(T/)|| 2 } = E{(z^[i] — z^[i]) T (z^[z] — z^[z])} = trace(P^[?]), which results 



in ( 34 1 again using the Comparison Lemma. 



Note that the upper bound (|34| is conditioned on the sampling intervals. Unfortunately, it is 
difficult to calculate E{trace(P|[i])} as a function of average sampling frequencies, which makes it 
hard to eliminate the conditional expectation. However, for the case where pi — ng, the upper 



bound (31 1 would also hold for the estimator in (33 1. This is indeed true because (33) is an optimal 



mean square error estimator. 

4 Applications to Networked Control 

In this section, we study networked control as an application of the proposed stochastic scheduling 
policy. Let us start by presenting the system model and the control law. We first present the results 



for impulsive controllers in Subsection 4.2 However, in Subsections 4.3 and|4.4| we generalize these 



results to pulse and exponential controllers. 
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4.1 System Model and Controller 

Consider the stochastic control system 

dze(t) = (-"/ e ze(t) + v e (t)) dt + a t dwt(t); z e (0) = z°, 



(35) 



where ze(t) G R and vt(t) G R, 1 < £ < L, are the state and control input of subsystem £. We assume 
that each subsystem is in feedback interconnection with a subcontroller governed by the control law 



(36) 



i=0 



where yf = z(Tf) + rii for all i G Z> and / : R — s- R U {±00} is chosen appropriately to yield 
a causal controller (i.e., f(t) = for all t < 0). For instance, using /(•) ~ S(-), where 5(-) is the 
impulse function (see [49[ p. 1]), results in an impulsive controller, which simply resets the state of 
its corresponding subsystem to a neighborhood of the origin characterized by the amplitude of the 
measurement noise whenever a new measurement is received. Without loss of generality, we assume 



that 



because the influence of the initial condition is only visible until the first sampling 



2.3 



instance XJj , which is guaranteed to happen in a finite time thanks to Assumption 
4.2 Performance Analysis: Impulsive Controllers 

In this sub sect ion, we present an upper bound for the closed-loop performance of subsystems de- 
scribed in (35 1 and controlled by an impulsive controller. In this case, for all t G [T/,T/ +1 ), the 
closed-loop subsystem £ is governed by 

dz e (t) = -jiZt(t) dt + a t dw e (t); zt{Tf) = -n\. 

The next theorem presents an upper bound for the performance of this closed-loop system which 
corresponds to the estimation error upper bound presented in Theorem |3.2| 



Theorem 4.1 Assume that subsystem £ : 1 < I < L , is described by \3!fy an d let the sequence 
of sampling instances {T/}°^ satisfy Assumption 2.3 Then, if ni < a/1/ '(2jg)o-£, the closed-loop 
performance of subsystem £ is bounded by 



E {*f(t)} < ^e-*"//' + fL (l - e -^i/h) . 



(37) 



otherwise, 



2jt 



,-2-ye/h 



Proof: Following the same reasoning as in the proof of Theorem 
derive 



3.2 



(38) 



for all t G [T/,T/ +1 ), we can 



-2-yet 



2li 



-2 7£ t\ 



(39) 



Let us assume that rji < y / l/(2^fe)ai. Hence, we get 

E {zj (t) I T? +1 - T() < nle-^^ + & (l - e-*»P^- 



since according to Lemma 3.1 (a), the right-hand side of (39 1 is a non-decreasing function of time 
whenever rje < ^l/(2jt)ae. Therefore, we get 



E{zKt)}=E{E{zm\TUi~n}} 



< E j^e-a-wWf+i-**) + ^ (l - e -27*(^ + i-^)) 



(40) 
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Using Lemma |3.l| (b) and Jensen's Inequality results in 
E jr^e- 2 ^!-^) + £ (l - 



< ^-M^-^} + ^ f 1 - e -*r^i-^} 



(41) 



Substituting (41 ) into (401 while using Theorem 2.6 we get the inequality in (37). For the case that 
the inequality rji < y // l/(2ji)ae is not satisfied, we can similarly derive the upper bound 

E {zj(t) I Tf +1 - T/} <rf e + ± (l - e -*»Ptx-^)) , 



which results in (38 1 by using Lemma 3.1 (b) and Jensen's Inequality. ■ 

Note that the closed-loop performance, measured as the variance of the plant state, is upper 
bounded by the plant and measurement noise variance. In the next two subsections, we generalize 
this result to pulse and exponential controllers. 

4.3 Performance Analysis: Pulse Controllers 

In this subsection, we use a narrow pulse function to approximate the behavior of the impulse 
function. Let us pick a constant p G R>o- For t G [T/,T/ +1 ), we use the control law 



«/(*) = 

whenever Tf + p < Tf +1 , and 



-yfr/e-™y(l - e~ w ), Tf < t < Tf + p, 
0, Tf + p<t<TL 1} 



(42) 



vi(t) = -yf lee -^/(l - e-T"), T t e <t< T/ +1 , 
otherwise. This controller converges to the impulsive controller as p tends to zero. 



Theorem 4.2 Assume that subsystem £, 1 < £ < L, is described by (35) and let the sequence of 
sampling instances {Tf}°^ Q satisfy Assumption 2.3 Then, the closed-loop performance of subsystem I 
is bounded by 



E{z 2 (t)} 



< 



r 2 

27f 



1 - P{Af < p} ' 



(43) 



Proof: To simplify the calculations, we introduce the change of variable z' e (t) — zt(t) + Ce(t) f° r au 
*e [T/,T/ +1 ), where 



t , e -itV- T V- e -~<e» 



\- e —W> 



+ n| I £^(l- e -^-^)), t€[Tf,T( + p), 

te[T( + p,T( +1 ). 



Now, using Ito's Lemma [44j p. 49], we get 

dz'z(t) = -j e z' e (t)dt + a e dw e (t); z' e {T[) = 
Hence, for all t G p; £ ,T/ +1 ), we get 

d 
in 

and as a result 

T 2 



I Af}<^ (i_ e -^ +m 



E{, 2 (t)}<E{^(l-e- 



27f A' 



■ 77 2 e~ 27fP + E{z 2 (T/)}P{A- < p}. 



(44) 
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Using Lemma |3.l| (b) and Jensen's Inequality, we can simplify ( 44 1 as 

1 

Therefore, we get 



W(*)} < S- (l - e_27i//£ ) + ^ e_27fP + E{^ 2 (^)}P{Af < p}. 



E{^ 2 W}<E 



' r ^-2 



fc=0 



2 7 £ 



1-e 



-lie/ ft 



[Af <p})' 



for all £ € [T/, and as a result 

oo p 2 



2 lt V 



+ % 2 e" 2 ^ 



[Af <p})' 



% V 2 ^ 



l-P{A^<p}- 



This concludes the proof. 



(45) 



Note that if p tends to zero in ( 45 ) , we would recover the same upper bound as in the case of the 
impulsive controller (38). This is true since limp^oPiA- < p} = assuming that the probability 
distribution of hitting-times of the underlying Markov chain is atom-less at the origin, which is a 
reasonable assumption when the Poisson jump rates are finite. 



4.4 Performance Analysis: Exponential Controllers 

In this subsection, we use an exponential function to approximate the impulse function. Let us pick 
a constant 8 € M >0 \ {"fe}- For all t £ [T/,T/ +1 ), we use the control law 



«<(*) = (7<-«)Bfe-^). 
This controller converges to the impulsive controller as 9 goes to infinity. 



(46) 



Theorem 4.3 Assume that subsystem I, 1 < I < L, is described by (35) and let the sequence of 
sampling instances {T? }™ satisfy Assumption 2.3 Then, the closed-loop performance of subsystem I 
is bounded by 



E{z 2 (t)} 



< 



+ °L ( i _ e -*r*/fA 



1 -E{e 



-28 A 1 . 



}' 



(47) 



Proof: Using the same argument as in the proof of Theorem |4.2[ we obtain 



E {zf(t) | A*} < 4 + & (l _ e -*H*V) + E{z 2 (T/)}e- 2 ^, 



for all t £ [T/,T/ +1 ), and as a result 



E |z|(t)} <^ 2 +E 



27£ 



1-e 



-27tA* 



E{z 2 (T/)}E{ £ 



Similar to the proof of Theorem 4.2 we can simplify this expression into (47) using Lemma 3.1 (b) 
and Jensen's Inequality. ■ 

Note that if 6 goes to infinity, we would recover the same upper bound as in the case of the 
impulsive controller since \img^ f+OD K{e~ 2eAi } = assuming that the probability distribution of 
hitting-times of the Markov chain is atom-less at the origin. Exponential shape of the control signal 
are common in biological systems such as in neurological control system [501. 
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Figure 2: An example of a networked system composed of decouple scalar subsystems (left) and 
non-scalar subsystems (right). 

5 Numerical Example 

In this section, we demonstrate the developed results on a networked system composed of L decoupled 
water tanks illustrated in Figure [2] (left) , where each tank is linearized about its stationary water 
level hg as 

dz t (t) = ~*f£-zt(t)dt + dvjt(ty, zi(0) = zj. 
a l y 2ti£ 

In this model, a' e is the cross-section of water tank £, ai is the cross-section of its outlet hole, and g 
is the acceleration of gravity. Furthermore, zi(t) and vi(t) denote the deviation of the tank's water 
level from its stationary point and the control input, respectively. Let the initial condition zf = 
as we assume that the tank's water level start at its stationary level. However, due to factors such 
as input flow fluctuations, the water level drifts away from zero. In the next subsection, we start by 
numerically demonstrating the estimation results for L = 2 water tanks. 



5.1 Estimation: Scalar Subsystem 

, = 1.00 m 2 , a x = 



0.20m 2 , a 2 = 0.10m 2 , g = 9.80m/s 2 , h t = 0.40m, 



Let us fix the parameters a[ = t 
and h 2 = 0.54 m. For these physical parameters, the water tanks can be described by (23) with 
71 = 0.7, 72 = 0.3, and <j\ = a 2 = 1-0. We sample these subsystems using the Markov chain in Q 
with m = 2L — 4. We assume that /itj(t) = /i^o + u,i(t) for all 1 < i < 4, where /i 2 e,o = 1 and 
Vtel—ifi = 10 for £ — 1,2. We are interested in finding itj(t), 1 < i < 4, in order to minimize the cost 
function 



J= lim e|- / Q.^eJx{t)dN 2 + Q.leJx{t)dN A 



u(t) T u(t)dt 



Using Corollary |2.2| we get 



ui(t,x) 




' -0.0228 










u 2 (t,x) 
u 3 (t,x) 










-0.0228 


-0.2272 



x(t) 


u^{t, x) 










-0.0272 





Figure [3] (upper-left) illustrates an example of the continuous-time Markov chain state x(t) and 
the sampling instances {T[}^1 of subsystems I = 1,2. Using (22), we get the average sampling 
frequencies f\ — 0.66 and f 2 — 0.83. Figure [3] (upper-right) shows the sampling instances when using 
a periodic scheduling policy with the same sampling frequencies as the average sampling frequencies 
of the optimal scheduling policy. Note that the optimal scheduling policy allocates the sampling 
instances according to the jumps between the states of the Markov chain. 

We can tune the average sampling frequencies of the subsystems by changing the design pa- 
rameters 1 < I < L. Table [TJ illustrates the average sampling frequencies of the subsystems 
versus different choices of the design parameters 1 < I < L. It is evident that when increasing 
(decreasing) ^ for a given £, the average sampling frequency of subsystem £ decreases (increases). 

Let us assume that estimator £ has access to state measurements of subsystem I according to ( 24 ) 
with measurement noise variance r\i = 0.3 for £ = 1,2. Figure [3] (lower-left) illustrates the estimation 
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Figure 3: An example of the state of the continuous-time Markov chain used in the optimal schedul- 
ing policy and its corresponding sampling instances for both subsystems (upper- left). Sampling in- 
stances for both subsystems when using a periodic scheduling policy (upper-right). Estimation error 
E{e^(i)} for 1000 Monte Carlo simulations when using the optimal sampling policy (lower- left) and 
the periodic sampling policy (lower-right). 



error variance E{ej(t)} for 1000 Monte Carlo simulations when using the optimal scheduling policy. 



The horizontal lines represent the theoretical upper bounds derived in Theorem 3.2 i.e., E{ef (i)} < 
0.64 and E{e|(t)} < 0.90. Note that the approximations of the estimation error variances would 
eventually converge to the exact expectation value as the number of simulations goes to infinity, and 
that the theoretical bounds are relatively close. Figure [3] (lower- right) illustrates the estimation error 
variance E{ef (t)} for 1000 Monte Carlo simulations when using the periodic scheduling policy that 
is portrayed in Figure [3] (upper-right) . Note that the saw-tooth behavior is due to the fact that the 
sampling instances are fixed in advance and they are identical for each Monte Carlo simulation. 



5.2 Estimation: Higher-Order Subsystems with Noisy State Measure- 
ment 

Let us focus on a networked system composed of only two subsystems where each subsystem is a 
serial interconnection of two water tanks. Figure (right) illustrates such a networked system. In 



Table 1: Example of average sampling frequencies. 



a 


6 


fx 


h 


0.1 


0.1 


0.8040 


0.8040 


0.5 


0.1 


0.6577 


0.8279 


1.0 


0.1 


0.4656 


0.8559 


2.0 


0.1 


0.0451 


0.9045 
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Figure 4: Estimation error E{||e£(t)|| 2 } for 1000 Monte Carlo simulations and its comparison to the 
theoretical results when &i = 2 for £ = 1, 2. 
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Figure 5: Estimation error E{||e£(i)|| 2 } for 1000 Monte Carlo simulations when using Kalman-filter 
based estimator. 



this case, subsystem £ can be described by (29) with 
A.= 



- (o<,T/Qf ,t) V 'g/( 2fe AT) o 
+(o<,t/o£ jT ) y/ g/ (2/if/r) - (o*,b/ <4, b ) V #/ ( 2 ^,b) 



where the parameters marked with T and B belong to the top and the bottom tanks, respectively. Let 
us fix parameters a' x T = a[ B = a 2 T = a 2 B = 1.00m 2 , cj^t = c^b = 0.20m 2 , a 2j T = a 2.B = 0.10m 2 , 
fti T = B = 0.40 m, and h 2 .T = h 2 -g = 0.54 m. 

Let us assume that estimator £ has access to the noisy state measurements of subsystem £ 
(with noise variance E{nfnf T } = 0.09/2x2) at sampling instances {T[}^ enforced by the opti- 
mal scheduling policy described in Subsection |5.1[ Figure [4] shows the estimation error variance 
E{|| ei(t) || 2 }. The horizontal lines in this figure show the theoretical bounds calculated in Theo- 
rem p3j i.e., E{||ei(t)|| 2 } < 2.05 and E{|)e2(<)|| 2 } < 2.21. In comparison with the scalar case in 
Figure |3| (lower-left), note that the bounds in Figure [4] are less tight. The reason for this is that the 
dimension of the subsystems are now twice the previous case. 

5.3 Estimation: Higher-Order Subsystems with Noisy Output Measure- 
ment 

In this subsection, we use output measurements yf = [0 l]ze.(Tf) + n\ for all i G Z>q, wh ere 
E{(?if) 2 } = 0.09 for £=1,2. We use the Kalman filter based scheme introduced in Subsection 3.4 for 



estimating the state of each subsystem. Figure [5] illustrates the estimation error variance E{||e^(i)|| 2 }. 
As mentioned earlier, it is difficult to calculate E{trace(P^ [i])} as a function of the average sampling 
frequencies and hence, we do not have any theoretical results for comparison. Note that the upper 
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Figure 6: Estimation error E{ej(t)} for 1000 Monte Carlo simulations over an ad- hoc networked 
system with the optimal sampling policy (left) and the periodic sampling policy (right). 



bound presented in Theorem 3.4 is only valid for a fixed sequence of sampling instances. This problem 
can be an interesting direction for future research. 



5.4 Estimation: Ad-hoc Sensor Network 

As discussed earlier, an advantage of using the introduced optimal scheduling policy is that we can 
accommodate for changes in ad-hoc networked systems. To portray this property, let us consider a 



networked system that can admit up to L = 70 identical subsystems described by (23) with 7^ = 0.3 
and (Ti = 1.0 for 1 < £ < 70. When all the subsystems are active, we sample them using the Markov 
chain in ^ with m = 2L = 140. We assume that ^i{t) — /li.o + u i(t) for 1 < i < 140, where 
A*2£.o = 10 and fi2i-i.o = 50 for 1 < £ < 70. In this case, we are also interested in calculating an 
optimal scheduling policy that minimizes 




x(t)dN 2 e + u(t) T u(t)dt 



(48) 



However, when some of the subsystems are inactive, we simply remove their corresponding nodes 
from the Markov chain flow diagram in Figure [I] (right) and set their corresponding terms in ( |48[ ) 
equal to zero. Let us assume that for t £ [0,5), only 30 subsystems are active, for t £ [5, 10), all 
70 subsystems are active, and finally, for t £ [10, 15], only 10 subsystems are active. Figure [6] (left) 
and (right) illustrate the estimation error variance E{e^(i)} for 1000 Monte Carlo simulations when 
using the optimal scheduling policy and the periodic scheduling policy, respectively. Since in the 
periodic scheduling policy, we have to fix the sampling instances in advance, we must determine the 
sampling periods according to the worst-case scenario (i.e., when the networked system is composed 
of 70 subsystems). Therefore, when using the periodic sampling, the networked system is not using 
its true potential for t £ [0,5) and t £ [10, 15], but the estimation error is fluctuating substantially 
over the whole time range. The proposed optimal scheduling policy adapts to the demand of the 
system, see Figure [6] (right). For instance, as shown in Figure [6] (left), when subsystems 31 and 32 
become active for t £ [5, 10), the overall sampling frequencies of the subsystems decreases (and, in 
turn, the estimation error variance increases), but when they become inactive again for t £ [10, 15], 
the average sampling frequencies increase (and, in turn, the estimation error variance decreases). 
Hence, this example illustrates the dynamic benefits of our proposed stochastic scheduling approach. 



5.5 Controller: Decoupled Scalar Subsystems 

In this subsection, we briefly illustrate the networked control results for L — 2 subsystems. Let the 
subsystems be described by (351 with 71 = 0.7, 72 = 0.3, and o\ = 02 = 1.0. Let us assume that 
controller £ has access to state measurements of subsystem £ according to (24) with measurement 
noise variance rji = 0.3 for £ = 1,2. We use the optimal scheduling policy derived in Subsection |5.1| 
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Figure 7: An example of state and control of the closed-loop subsystems when using the impulsive 
controller (left) and the exponential controller with 9 = 10 (right). 
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Figure 8: Closed-loop performance measure E{z|(£)} for 1000 Monte Carlo simulations when using 
the impulsive controller (left) and the exponential controller (right). 



for assigning sampling instances. Figure [7] (left) and (right) illustrate an example of the state and 
the control signal for both subsystems when using the impulsive and exponential controllers (with 
9 = 10), respectively. Note that in Figure [7] (left), the control signal of the impulsive controller only 
portrays the energy that is injected to the subsystem (i.e., the integral of the impulse function) and 
not its exact value. Figure [8] (left) and (right) show the closed-loop performance E{z^(i)} when 
using the impulsive and exponential controllers, respectively. The horizontal lines illustrate the 
theoretical upper bounds derived in Theorem |4.1| Note that the exponential controller gives a worse 
performance than the impulse controller. This is normal as we design the exponentials only as an 
approximation of the impulse train. 

5.6 Controller: Coupled Scalar Subsystems 

Consider a networked system composed of L — 70 interconnected subsystems, where subsystem £, 
1 < i < 70, can be described by 

faZt(t) = 0-l(z mo d(e-i,L)(t) - zi(t)) + 0.1(z mod ( i+1}L )(t) - zt(t)) + Vi(t) + wt(t); z e (0) = 0. 

with notation mod(i, j) = i — for any i e Z and j € Z>o- In this model, zt(t), ve(t), and we (t) 

respectively denote the state, the control input, and the exogenous input. Each subsystem transmits 
its state measurement over the wireless network at instances {Tf }gi to its subcontrollcr. Hence, 



21 




-0.4 1 1 1 1 1 • 1 

5 10 15 20 25 30 




5 10 15 20 25 30 

Times (sec) 



Figure 9: An example of state and control sig; 
periodic sampling policy (right). 
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using the optimal sampling policy (left) and the 



at any time t E R>o, subcontroller £ has access to the state measurements 2^(T~^) where recalling 

from the earlier definitions Mf = maxji > 1 1 T? < t}. Each subcontroller simply implement the 
following decentralized proportional-integral control law 

Vi(t) = -l.2z t {T l Mt ) - 0.3 / Zi (T e Mt )dT. 
* Jo 

We sample the subsystems using the Markov chain in ^ with m = 2L = 140. We assume that 
fXiit) = fjbifi + Ui{t) for 1 < i < 140, where H2e.fi = 10 and //^-l.o = 70 for 1 < I < 70. Let 
us consider the following disturbance rejection scenario. We assume that we(t) = for I 7^ 4,26, 
Wi{t) = step(i), and W26(t) — — 0.4 step(t — 15), where step : R — > {0, 1} is the heaviside step function 
(i.e., step(i) = 1 for t £ ]R>o and step(t) = 0, otherwise). Let us denote t G [0, 15) and t E [15, 30] as 
the first phase and the second phase, respectively. During each phase, we find the infinite horizon 
optimal scheduling policy which minimizes 

f 1 r T 70 

J — lim E < — / V & eJ 1 x(t)dN 2i + u(t) T u(t)dt 
T ^°° [ TJ o 

We fix = 10 for £ = 4 over the first phase and for £ = 26 over the second phase. In addition, 
we fix £f = 20 for £ = 3, 5 over the first phase and for £ = 25,27 over the second phase. Finally, 
we set & = 30 for the rest of the subsystems. This way, we can ensure that we more frequently 
sample the subsystems that are most recently disturbed by a nonzero exogenous input signal (and 
the ones that are directly interacting with them). Figure [9] (left) and (right) illustrate an example 
of the system state and control input when using the described optimal scheduling policy and the 
periodic scheduling policy, respectively. For the periodic scheduling policy, we have fixed the sam- 
pling frequencies according to the worst-case scenario (i.e., the average frequencies of the optimal 
scheduling policy when £^ = 10 for all 1 < £ < 70 corresponding to the case where all the subsystems 
are disturbed). As we expect, for this particular example, the closed-loop performance is better 
with the optimal scheduling policy than with the periodic scheduling policy. This is indeed the case 
because the optimal scheduling policy adapts the sampling rates of the subsystems according to their 
performance requirements. 



6 Conclusions 

In this paper, we used a continuous-time Markov chain to optimally schedule the measurement and 
transmission time instances in a sensor network. As applications of this optimal scheduling policy, 
we studied networked estimation and control of large-scale system that are composed of several 
decoupled scalar stochastic subsystems. We studied the statistical properties of this scheduling 
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policy to compute bounds on the closed-loop performance of the networked system. Extensions of the 
estimation results to observable subsystems of arbitrary dimension were also presented. As a future 
work, we could focus on obtaining better performance bounds for estimation and control in networked 
system as well as combining the estimation and control results for achieving a reasonable closed-loop 
performance when dealing with observable and controllable subsystems of arbitrary dimension. An 
interesting extension is also to consider zero-order hold and other control function for higher-order 
subsystems. 
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